Skip to content

docs(ops): add retention-gap recovery runbook - #181

Open
Hustler490 wants to merge 1 commit into
wraith-protocol:developfrom
Hustler490:docs/retention-gap-recovery
Open

docs(ops): add retention-gap recovery runbook#181
Hustler490 wants to merge 1 commit into
wraith-protocol:developfrom
Hustler490:docs/retention-gap-recovery

Conversation

@Hustler490

@Hustler490 Hustler490 commented Oct 2, 2026

Copy link
Copy Markdown

Closes #159

What

The deployment guide documents Soroban RPC retention windows and the mitigation options, but stops there. It never tells an operator what to actually do when a scanner is offline past the boundary. This adds a dedicated recovery runbook.

Changes

  • guides/ops/retention-gap-recovery.mdx (new) - step-by-step procedure covering:
    • Detection - compare lastProcessedLedger to the network tip and classify the gap against the ~17,280-ledger window; confirm via retention_window_exceeded / RetentionExceededError.
    • Checkpoint handling - stop the scanner, capture lastProcessedLedger before and after, never advance the checkpoint past unpersisted events.
    • Archive replay - full-history node with fallbackPolicy: "archive_node".
    • Dual-provider recovery - fail over to an independent provider for the missed range.
    • Horizon fallback - best-effort reconstruction when no archive/secondary covers the range.
    • Safe rescan + deduplication - stable event identity (contract_id, ledger, tx_hash, event_index) with a unique constraint and ON CONFLICT DO NOTHING upsert.
    • Alert thresholds - approaching-boundary (medium) and past-boundary (critical) Prometheus rules.
    • Operator evidence checklist and a post-recovery verification list.
  • guides/ops/monitoring-and-on-call.mdx - links the runbook from Playbook 3 (Indexer Stall) and Cross-References.
  • docs.json - registers the page in the Operations navigation group.

Validation

  • node localization-qa.cjs ? 0 issues (new page recognised as a canonical page).
  • node scripts/check-nav-coverage.mjs ? 72 pages checked, 72 nav entries verified.
  • docs.json parses as valid JSON.

@truthixify

truthixify commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

The replacement removes the fake CLI commands, but several examples still use APIs that do not exist. fetchAnnouncementsStream is a Stellar entry-point export and takes (chain, options); it has no contractIds, retention, or fallbackPolicy. Horizon has no getTransactions helper, and announcements do not expose event_index. Please align the runbook with the current SDK and existing stellar_network_ledger_height metric.

@truthixify truthixify mentioned this pull request Oct 2, 2026
Closed
4 tasks done
@Hustler490
Hustler490 force-pushed the docs/retention-gap-recovery branch from 34b215a to d9f412d Compare October 2, 2026 15:39
@truthixify

truthixify commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

The new head still contains the same invalid SDK and Horizon examples. Please address the API, event identity, and metric feedback above before requesting another review.

@Hustler490
Hustler490 force-pushed the docs/retention-gap-recovery branch from d9f412d to c111d45 Compare October 4, 2026 15:58
@Hustler490
Hustler490 requested a review from truthixify as a code owner October 4, 2026 15:58
@truthixify

truthixify commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

The latest push replaced this PR with an unrelated localization commit and removed the retention recovery runbook. Please restore the correct branch on top of current develop, then push the runbook changes again.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@truthixify truthixify Awaiting requested review from truthixify truthixify is a code owner

At least 0 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Wave 9] Add a retention-gap recovery runbook for operators

3 participants

@Hustler490 @truthixify @Vivian-04